Live tests for the Python and asyncio docs code samples - #241

What

Adds docs-snippets/, a pytest suite that runs the code samples of the Python and asyncio SDK docs against the real PubNub service and asserts the documented effect. The docs embed the snippet regions with EmbeddedCode from this repo (refs/heads/master/docs-snippets/...). It follows the JavaScript suite (pubnub/javascript PRs 538 and 539).

Two commits:

1. ci: remove the cross-workflow cancel step. Removes the four fast-jobs-failure steps from run-tests.yml and run-validations.yml. One failing fast job cancelled every workflow of the commit, so the first attempt of other workflows ended cancelled.
2. test: add live docs snippet tests. The suite, its workflow and the setup.py exclude.

What is covered

• 311 snippet ids in 193 steps: 183 live ids in 109 sync steps, 125 live ids in 84 asyncio steps, 3 compile-only
• Areas: publish, signal, fire, subscribe (new and legacy), presence, message persistence, message actions, channel groups, files, mobile push, App Context, Access Manager (v2 and v3), encryption, configuration, misc
• Not covered: signature blocks, response tables, interactive examples. The Python SDK has no DataSync API, so there is no DataSync role

How it stays honest

• Assertions sit outside the snippet markers. A marker lint enforces it, so the region stays what the docs show
• A step fails when its snippet did not reach snippet.end or when it made no assertion
• Every step has its own run-suffixed names, so a client that lingers in presence cannot affect the next step
• No skips, no xfail, no fixed sleeps (eventually for reads after writes), subset runs are refused in CI
• Keys and tokens are masked in output and reports, the janitor revokes application-level Access Manager grants and removes leftovers
• coverage.json lists every snippet id, python -m harness.cli manifest --check fails when it is stale
• python -m harness.audit lists assertions per step, one log file per step is written to report/<shard>/steps/

Latest live results (open and pam keysets)

• open shard: 160 passed, 4 failed
• pam shard: 29 passed
• The 4 failures are real SDK bugs and fail on purpose, nothing was weakened: get_message_actions ignores end when start is set (pubnub/endpoints/message_actions/get_message_actions.py:50), and PubNubSubscriptionSet.unsubscribe() leaves the channels subscribed (pubnub/models/subscription.py:162-182), each in sync and asyncio. The nightly stays red until these are fixed or the expectation changes. More SDK findings are in the follow-up message to the maintainers

Workflow

.github/workflows/docs-snippets.yml: a changes job, a dry job (no network, Python 3.9 to 3.13), a live job (open shard then pam shard in series under one concurrency group, Python 3.11) and a gate job to require in branch protection. Triggers: pull request, push to master, nightly, manual. Fork PRs skip the live job.

Needed before the live job can run

Secrets: SDK_DOCS_OPEN_PUBLISH_KEY, SDK_DOCS_OPEN_SUBSCRIBE_KEY, SDK_DOCS_PAM_PUBLISH_KEY, SDK_DOCS_PAM_SUBSCRIBE_KEY, SDK_DOCS_PAM_SECRET_KEY. The open keyset needs Message Persistence with Delete-From-History, Presence with a * tracking rule and legacy SDK tracking, App Context with events, Files, Mobile Push and Stream Controller with wildcard subscribe. The pam keyset needs Access Manager with token revocation. Without a GitHub Environment with required reviewers, anyone with push access can read the secrets through a pull request.

Not changed

pubnub/, tests/, examples/ and the version are untouched. Run it locally with cd docs-snippets && python -u -m harness.cli dry (no keys) or python -u -m harness.cli run. See docs-snippets/README.md.

🤖 Generated with Claude Code